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Abstract 

In this note I go through the 'proof of frequentistic confidence 
intervals and show what it logically implies concerning the value of a 
physical quantity given an experimental observation (nothing). 



". . . to emancipate us from the 
capricious ipse dixit of authority" 
(John Henry Newman) 

1 Introduction 

The construction of frequentistic confidence intervals is the statistical tool 
adopted in all 'conventional statistics' jlj books and lecture notes in order 
to provide the result on the value of a physical quantity resulting from a 
measurement. Though many physicists are aware of the unsuitability of 
the 'prescription' to handle critical cases that occur in frontier science (and 
the troubles are also known to the supporters of the prescription: see P], 
references therein and follows up), it seems they do not always recognize the 
real reason the prescription fails. 

* Based on a lectures to graduate students at the University of Rome "La Sapienza" 
(May, 4 2005 and May, 8 2006). 



In the past I am always been reluctant to go through the details of the 
definition of the frequentistic confidence intervals, simply because I thought 
that, once one has realized that: 

• the outcome of these methods is usually misleading (in the sense that 
the common interpretation differs from what they are supposed to 
mean, see e.g. Ref. 1^); 

• the resulting 'confidence' intervals can often come out absurd (see e.g. 
Ref. 0, sections 1.7 and 10.7); 

• the celebrated 'frequentistic coverage' does not do its job in the im- 
portant cases of interest in frontier physics (see Ref. 13!, section 10.7); 

then there is little more to do, apart from looking for something better. 
I admit I have been naive. In fact I have met too many students and 
colleagues touched by the above arguments but still not fully convinced, 
because impressed by the names given to the criticized methods (something 
that is classical, rigorous and exact cannot be wrong!) and the authority of 
books and publications that use and recommend them.^ 

The aim of this note is to go through the definition of the confidence 
interval as one finds it in books and lecture notes^ and to show why that 
reasoning yields no new information concerning the value of the unknown 
quantity of interest and about which one wants to gain some confidence 
based on experimental observations. 



found a curious analogy between the learning of fundamental concepts of physics and 
the learning of statistical methods for data analysis, as they are taught in the standard 
physics curriculum. It is a matter of fact that many concepts of physics are not easy at 
all to be grasped by students (and not not only by students!). After some time, students 
assume the habit to learn at least the practical formulas, trusting teachers and text books 
as far as correctness and deep meaning of the difficult concepts are concerned. When they 
first learn frequentistic statistics applied to data analysis, presented in the usual shifty 
way we know, the students tend to act in the same way, thinking the things must be 
difficult but correct, because granted by the teachers, who in most case just repeat the 
lesson they learned but tacitly hold the same doubts of the pupils. The small 'Socratic 
exchange' by George Gabor reported in Ref. i3j might be enlightening. 

^I avoid to give a particular reference. Everyone can check his/her preferred text and 
see how confidence intervals are presented. 
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2 From the probability interval of the estimator 
to the confidence about the true value 9 



Here is, essentially, the basic reasoning that introduces confidence intervals. 

1. Let 6 be the true value of the quantity of interest and 6 its 'estimate' 
(an experimental quantity, probabilistically related to 9). 

2. We assume to know, somehow,^ the probability density function (pdf) 
of for any given value 6, i.e. f{9 \ 9). The knowledge of f{6 \ 6) allows 
then us to make probabilistic statements about ^, for example that 9 
will occur in the interval between 6 — A^i and 9 + A02 '■ 

P{9- Mi<9 <9 + M2) = a. (1) 

For example, if f{9 \ 9) is a Gaussian and A^i = A02 = then a = 
68.3% (hereafter it will be rounded to 68%). 



3. To establish the confidence interval about 9, it is said, we invert the 
probabilistic content of ^\)■. 

from [0 - ^0, < 9 i 9 < 9 + A9, 

\ 9 + A92 < 9 \ 9 > 9 - A92 ' 

thus rewriting ^ as 

P{9-A92<9<9 + A9i) = a. (3) 



Then, it usually follows some humorous nonsense to explain what © is not 
and what, instead, should be. Some say that this formal expression does 
not represent a probabilistic statement about 9, because 9 is not a random 
variable, having a well defined value, although unknown. Instead, it is said, 
9i = 9 — A92 and 92 = 9 — A9i are random variables. Then, the meaning of 
^ is not probability that 9 lies inside the interval, but rather probability 
that the interval [^1,^2] encloses ^, in the sense ('frequentistic coverage') 
that if we repeat an infinite number of times the experiment, the true value 
9 (that is always the same) will be between 9i and 92 (that change from time 
to time) in a fraction a of the cases; i.e. in each single measurement the 
statement 9i < 9 < 92 has a probability a of being true. (And all people of 

skip over the fact that in most cases f{9 \ 9) is not reaUy obtained by past frequencies 
of 9 for some fixed value of 9, as the frequency based definition of probability would require. 
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good sense wonder what is the difference between the latter statement and 
saying that a. is the probabihty that d is between Q\ and 02-) 

Anyhow — and more seriously — , besides all this nonsense, what matters 
is what finally remains in the mind of those who learn the prescription and 
how Eq. is used in scientific questions. If a scientist knows /(6' | 6*) and 
observes 0, then he/she feels authorized by Eq. Q to he confident^ with 
confidence level a, that the unknown value of is in the range Q — A^2 < 
< ^ + A^i. And it is a matter of fact that, in practice, all users of the 
prescription consider Eq. as a probabilistic statement about a, i.e. they 
feel confident that Q is in that interval as he/she is confident to extract a 
white ball from a box that contains a fraction a of white balls. ^ 

3 What do we really learn from Eqs. dH)— (jSI)? 

Let us now go through the details of the previous 'proof and try to un- 
derstand what we initially knew and what we know after the 'probabilistic 
content inversion' provided by Eq. (jSJ. In particular, we need to understand 
what we have really learned about the unknown true value Q as we went 
through the steps (P)-®- 

Given the general assumptions, the statement is certainly correct. 
One would argue whether f(Q \ 6) is indeed the 'right' pdf, but this is a 
different story (the issue here is just logic, as we are only interested in logical 
consistency of the various statements). Much awareness is gained about 
what is going on in steps Q-©, if we rewrite Eq. stating explicitly 
the basic assumption as a explicit condition in the probabilistic statement, 
and distinguishing the name of variable from its particular numerical value, 

*What happens is that the poor teacher (see footnote 1) at the end of the day is forced 
to tell that Eq. is 'in practice' a probabilistic statement about 0, perhaps adding that 
this is not rigorously correct but, essentially, 'it can be interpreted as if. However, this 
is not just an understandable imprecision of the 'poor teacher', in conflict between good 
sense and orthodoxy [T] . For example, we read in Ref. 15] (the authors are influential 
supporters of the use frequentistic methods in the particle physics community): 

When the result of a measurement of a physics quantity is published as 
R — Ro i o-Q without further explanation, it simply implied that R is a 
Gaussian-distributed measurement with mean Ro and variance ctq. This 
allows to calculate various confidence intervals of given "probability" , i.e. 
the "probability" P that the true value of R is within a given interval. 

(The quote marks are original and nowhere in the paper is explained why probability is 
in quote marks.) 
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indicating the former with a capital letter, as customary: 



P[e-Aei<Q<0 + M2 I f{d \d)] = a, (4) 

that, for a Gaussian distribution of [indicated by the shorthand ~ 
AA(^, fj)] and for A^i = /S.62 = cr, becomes 

P[9-a<e<e + a\e^M{e,a)] = 68%: (5) 

if we know the value of 9 and the standard deviation of the distribution we 
can evaluate the probability that B shall occur in the interval [6 — a, 9 + cr], 
where 'to know' means that 9 and cr have some numeric values, e.g. 9 = 5 and 
o" = 2. I find particularly enlightening to use, for a while, these particular 
values of fj, and a, rewriting Eq. Q as 

P[5-2 < e < 5 + 2 I e ~AA(5,2)] = 68%. (6) 

Obviously, knowing Q ~ AA(5, 2), we can write an infinite number of proba- 
bilistic statements. In particular 

P[e < 5-2 I e ~A/'(5,2)] = 16% (7) 

P[e > 5 + 2 I e ~ A/'(5, 2)] = 16%, (8) 

i.e. 

P[e + 2 < 5 I e ~ A^(5, 2)] = 16% (9) 

P[e - 2 > 5 I e ~ A/'(5,2)] = 16%, (10) 

from which we have^ 

P[e-2<5<G + 2 I e~A/'(5,2)] = 68%. (11) 

Obviously, this expression is valid for any known value of 9 and a: 

P[e-a <9 <e + a \er^M{9,a)] = 68%. (12) 

read once in a frequentistic book something like this: "i/ you do not trust logic, 
prove it with a Monte Carlo" . These are the two Unes of R code needed to 'prove' by 
Monte Carlo the equality of Eqs. (|SJ and IIH : 
X ^ rnormClOOOO, 5, 2) 

length(x[ (5-2) < x & x < (5+2) ]) == length(x[ (x-2) < 5 & 5 < (x+2) ]) . 
Similarly, for an asymmetric interval, e.g. AOi = 2 and A62 ~ 3, we have 
length(x[ (5-2) < x & x < (5+3) ]) == length(x[ (x-3) < 5 & 5 < (x+2) ]) . 
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The fact that we have replaced the numbers 5 and 2 by the generic symbols 
9 and a changes nothing about meaning and possible use of 1)121) : just a 
rephrasing of a probabilistic statement about Q! 

This is also the meaning of the 'probabilistic content inversion' ©-Q: a 
simple rephrasing of a probabilistic statement about (that indeed should 
be 0) under the assumption that we know the value of 9 and the pdf of '0' 
around 9: 

P[9-A9i<e<9 + A92 I f{0 \e)] = a (13) 

equivalent to 

P[e - < ^ < e + A6'i I /(^ 1 6')] = a (14) 

Therefore, there is no doubt that ()14)) follows from ()13)) . The question is 
that this is true if 9, A9i and A02 are real numbers, whose values, together 
with the knowledge of f{9 \ 9), allow us to calculate a. However, rephrasing 
the probabilistic statement concerning the possible observation Q given a 
certain 9 does not help us in solving the problem we are really interested 
in, i.e. to gain knowledge about the true value and to express our level of 
confidence on it, given the experimental observation 9. 

What we are really looking for is, indeed, P{9i < Q < 92 \ 9). But this 
can be only achieved if we are able to write down, given the best knowledge of 
the physics case, the pdf f{9 \ 9). Pretending to express our confidence about 
without passing through f{9\9) is pure nonsense, based on a proof that 
reminds the 'game of the three cards' proposed by con artists in disreputable 
streets. Now, it is a matter of logic that the only way to go from f{9 \ 9) 
to f{9 I 9) is to make a correct 'probability inversion', following the rules 
of probability theory, instead of that shameful outrage against logic. The 
probabilistic tool to perform the task is Bayes' theorem, by which it is 
possible to establish intervals that contain the true value at a given level 
of probability (meant really by how much we are confident the true value is 
in a given interval!). It is easy to show that, under well defined conditions 
that often hold in routine applications, the interval calculated at a given 
level of probability Op is equal to 'confidence interval' calculated with a 
'confidence level' acL, if numerically^ acL = otp (see e.g. Ref. ^3,). It is not 

® Those interested in Bayesian/frequentistic comparisons might give a look at Ref. 
Personally, as explained in Ref. ^ (footnote 18 of p. 229), I dislike the quantitative com- 
parisons of Bayesian and frequentistic methods to solve the same problem simply because 
quantitative comparisons assume that we are dealing with homogeneous quantities, while 
frequentistic CL's and Bayesian probability intervals are as homogeneous as apples and 
tomatoes are. (This is also the reason I used here two different symbols, acL and Op.) 
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a surprise, then, that the confidence interval prescriptions yield quite often 
a correct result, but they might also miserably fail, especially in frontier 
physics applications. 

4 Conclusions 

The proof of frequentistic 'confidence intervals' is sterile and there is no 
logical reason why one should attach a 'level of confidence' to the intervals 
calculated following that prescription. Paraphrasing a sentence of the same 
author of the opening quote, they are called confidence intervals by their 
advocates because they provide confidence in no other way. 
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